关键词: 文本语义去重, 关键词抽取, 文本匹配, 图表示, 图注意力网络

Abstract: There are a large number of redundant texts with the same or similar semantics in the network. Text de-duplication can solve the problem that redundant texts wasted storage space and reduce unnecessary consumption for information extraction tasks. Traditional text de-duplication methods mostly rely on literal information, and could not capture the interaction information between sentences that are far away in long text, so the de-duplication effect is not ideal. To solve the problem of text semantic de-duplication, a semantic de-duplication algorithm based on keyword graph representation was proposed. First, the text pair was represented as a graph with the keyword phrase as the vertex by extracting the semantic keyword phrase from the text pair. Secondly, the nodes were encoded in various ways, and then the Graph Attention Network (GAT) was used to learn the relationship between nodes to obtain the vector representation of text pairs, and to judge whether the text pairs were semantically similar. Finally, the similar text was de-duplicated according to the text pair’s semantical similarity. Compared with the traditional methods, it can effectively use the semantic information of the text, and through the graph structure, it can connect the distant sentences in the long text through the co-occurrence relationship of keyword phrases to increase the semantic interaction between different sentences. Experiments show that the proposed algorithm performs better than the traditional algorithms, such as Simhash, BERT (Bidirectional Encoder Representation from Transformers) fine-tuning and CIG (Concept Interaction Graph), on both CNSE (Chinese News Same Event) and CNSS (Chinese News Same Story) datasets. The F1 score in the CNSE dataset reaches 84.65%, and the F1 score in the CNSS dataset reaches 90.76%, indicating that the proposed algorithm can effectively improve the effect of text de-duplication tasks.

Key words: text semantic de-duplication, keyword extraction, text matching, graph representation, Graph Attention Network (GAT)